Apparatus and method for synthesizing multi-dimensional texture

ABSTRACT

A multi-dimensional texture synthesis apparatus includes a generator to generate a reference multi-dimensional texture including a codebook and a reference index image, the codebook including a set of color information of the same-position pixels of the reference texture images, and the reference index image including a set of indexes for addressing the codebook, a quantizater to quantize the reference multi-dimensional texture at one or more quantization levels to generate a quantized codebook and a quantized reference index image, a synthesizer to synthesize a new index image having a specified size from the quantized reference index image using a hierarchical structure of the reference multi-dimensional texture corresponding to the one or more quantization levels, and a generator to generate the new multi-dimensional texture by combining the new index image with the quantized codebook.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of application Ser. No. 10/792,842, filed Mar. 5,2004, and claims the benefit of Japanese Patent Applications No.2003-062180, filed Mar. 7, 2003; and No. 2004-059407, filed Mar. 3,2004, all of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image synthesis suitable for highquality texture mapping in the field of three dimension computergraphics, particularly to a multi-dimensional texture synthesisapparatus to synthesize a texture image of an arbitrary size accordingto conditions such as a viewpoint direction and a light sourcedirection, and a multi-dimensional texture synthesis method.

The present invention relates to a data structure for effectivelystoring and managing a number of texture images to various kinds ofrecording media.

2. Description of the Related Art

The posture of a three-dimension object, that is, the shape of thethree-dimensional object and the surface color and quality impressionthereof vary based upon a direction (a viewpoint direction) watching thethree-dimensional object and a direction (a light source direction)irradiating a light beam. In a field of three-dimensional computergraphics, the surface of three-dimensional object is divided into anumber of two-dimensional plains called polygons, the drawing isexecuted every polygon to create a two-dimensional image to be used as adisplay image of three-dimensional object.

The posture of the three-dimensional object when the viewpoint directionand light source direction vary can be expressed by changing a directionof display polygon (three-dimensional posture) and the opticalcharacteristic of the polygon (brightness) along with the viewpointdirection and light source direction. However, computation of a relationbetween a normal defining a direction (three-dimensional posture) of thepolygon that is a drawing unit and a viewpoint direction or a lightsource direction needs a vector operation of floating point precision.This results in increasing a computing cost and complicating practicaluse in terms of hardware and software.

In late years, a scheme for synthesizing a texture image of an arbitrarysize by small texture images is proposed (Li-Yi Wei, Marc Levoy, “FastTexture Synthesis Using Tree-structured Vector Quantization”,Proceedings SIGGRAPH 2000, pp. 479-488, and Michael Ashikhmin,“Synthesizing Natural Textures”, 2001 ACM Symposium on Interactive 3DGraphics, pp. 217-226). This scans a texture image of an arbitrary sizereferring to an original small texture image and synthesizes a textureimage near to the original image.

In addition, there is proposed a scheme for preparing a target textureimage other than the referring small texture image and synthesizing animage having the design and pattern of the referring texture image andresembling the materials of the referring small texture image.

However, such a conventional texture image synthesis technologysynthesizes merely a simple texture image, and cannot synthesize in alump a number of texture images concerning multiple viewpoints andmultiple light sources.

As described above, when the optical characteristic of the objectsurface varying according to a viewpoint direction and a light sourcedirection is expressed in texture, a large number of texture images thatdiffer in the viewpoint direction and light source direction arenecessary, resulting in enormous computing quantity.

BRIEF SUMMARY OF THE INVENTION

It is an object of the present invention to provide an apparatus andmethod to synthesize from a number of texture image data a texture imageof an arbitrary size which can express realistically the opticalcharacteristic of the object surface varying according to a light sourcedirection and a viewpoint direction.

According to an aspect of the invention, there is provided amulti-dimensional texture synthesis apparatus to synthesize a newmulti-dimensional texture using, as one or more reference textureimages, one or more texture images captured or drawn on a plurality ofdifferent conditions, comprising: a reference multi-dimensional texturegenerator to generate a reference multi-dimensional texture including acodebook and a reference index image, the codebook including a set ofcolor information of the same-position pixels of the reference textureimages, and the reference index image including a set of indexes foraddressing the codebook; a reference multi-dimensional texture quantizerto quantize the reference multi-dimensional texture at one or morequantization levels and generate a quantized codebook and a quantizedreference index image; an index image synthesizer to synthesize a newindex image having a specified size from the quantized reference indeximage using a hierarchical structure of the reference multi-dimensionaltexture corresponding to one of the quantization levels; and a newmulti-dimensional texture generator to generate the newmulti-dimensional texture by combining the new index image with thequantized codebook.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a block diagram of a multi-dimensional texture synthesisapparatus according to a first embodiment of the present invention;

FIG. 2 is a block diagram of a multi-dimensional texture synthesisapparatus according to modification of the first embodiment;

FIG. 3 is a flowchart representing a series of steps of the processexecuted with the multi-dimensional texture synthesis apparatusaccording to the first embodiment;

FIG. 4 is a flowchart for generating a reference multi-dimensionaltexture and a target multi-dimensional texture;

FIG. 5 is a diagram for explaining a method of acquiring a texture imageon various conditions (viewpoint/light source);

FIG. 6 is a diagram showing an example of data structure of themulti-dimensional texture;

FIG. 7 is a diagram for explaining extraction of color information ofthe identical pixel from a plurality of reference texture images;

FIG. 8 is a diagram showing a form of multi-dimensional pixel data;

FIG. 9 is a flowchart of a reference multi-dimensional texturequantization;

FIG. 10 shows a configuration of layered multi-dimensional texture;

FIG. 11 is a flowchart of index image synthesis;

FIGS. 12A and 12B are diagrams for explaining a similar pixel search;

FIG. 13 is a diagram for explaining a similar pixel search in hierarchyh;

FIG. 14 is a block diagram of a multi-dimensional texture synthesisapparatus according to the second embodiment of the present invention;

FIG. 15 is a block diagram of a multi-dimensional texture synthesisapparatus according to modification of the second embodiment;

FIG. 16 is a flowchart showing a series of steps of a process executedwith the multi-dimensional texture synthesis apparatus according to thesecond embodiment;

FIG. 17 is a diagram for explaining a similar pixel search in hierarchyh;

FIG. 18 is a flowchart of index image synthesis according to the thirdembodiment of the present invention;

FIG. 19 is a diagram for explaining resemblance evaluation of a pixel;

FIG. 20 is a block diagram of a multi-dimensional texture synthesisapparatus according to the fourth embodiment of the present invention;

FIG. 21 shows a process flow of the multi-dimensional texture synthesisapparatus according to FIG. 20;

FIG. 22 shows a process flow of a multi-dimensional texture generator;

FIG. 23 is a diagram to explain a multi-dimensional texture generationmethod;

FIG. 24 shows a basic data structure of a multi-dimensional texture;

FIG. 25 shows a format example of multi-dimensional pixel data;

FIG. 26 shows a process flow of a multi-dimensional texture quantizer;

FIG. 27 shows a multi-dimensional texture with a hierarchical structure;

FIG. 28 shows a process flow of an index image composition unit;

FIG. 29 is a diagram to explain a method of a similar pixel search;

FIG. 30 is a diagram of to explain an index image synthesis method withthe use of an approximate evaluation value;

FIG. 31 is a diagram to explain data included in a control image;

FIG. 32 is a diagram to explain a synthesis method to refer to aplurality of materials based on a control image;

FIG. 33 is a diagram to explain a method of searching for an optimumindex in referring to a plurality of materials;

FIG. 34 shows a block diagram of a multi-dimensional texture synthesisapparatus having a multi-dimensional texture converter according to thefifth embodiment of the present invention;

FIG. 35 shows a process flow of a multi-dimensional texture synthesisapparatus according to FIG. 34;

FIG. 36 is a diagram showing an example of a blending process of aplurality of materials; and

FIG. 37 is a diagram to explain unionization of a codebook to bereferred to.

DETAILED DESCRIPTION OF THE INVENTION

There will now be described an embodiment of the present invention inconjunction with the accompanying drawings.

First Embodiment

FIG. 1 is a diagram of configuration of a multi-dimensional texturesynthesis apparatus according to the first embodiment of the presentinvention.

The multi-dimensional texture synthesis apparatus comprises a referencemulti-dimensional texture generator 101, a reference multi-dimensionaltexture quantizer 102, an index image synthesizer 103, a newmulti-dimensional texture generator 104 and a new texture imageconverter 105. The reference multi-dimensional texture generator 101inputs one or more reference texture images captured or drawn with aplurality of different conditions and generates a referencemulti-dimensional texture image by gathering color information items ofthe same-position pixels. The reference multi-dimensional texturequantizer 102 subjects the generated reference multi-dimensional textureto vector quantization at one or more quantization levels to generate acodebook. There will be later described a method of generating areference multi-dimensional texture, a data structure of themulti-dimensional texture and a vector quantization method.

The index image synthesizer 103 is supplied with a quantized referencemulti-dimensional texture and synthesizes a new index image of anarbitrary size by repeating a similar pixel search based on thequantized reference multi-dimensional texture. A method of synthesizinga new index image is later described in detail. The newmulti-dimensional texture generator 104 combines the new index imagegenerated by the index image synthesizer 103 with a codebook generatedby the reference multi-dimensional texture quantizer 102 to generate anew multi-dimensional texture according to the data structure of themulti-dimensional texture. The new texture image converter 105 issupplied with the new multi-dimensional texture and outputs a textureimage corresponding to a specific acquisition condition or a generationcondition.

FIG. 2 shows a configuration of a multi-dimensional texture synthesisapparatus concerning a modification of the first embodiment. In thismodification, like reference numerals are used to designate likestructural elements corresponding to those in FIG. 1 and any furtherexplanation is omitted for brevity's sake. This apparatus comprises aconfiguration that the new texture image converter 105 is removed fromthe apparatus of FIG. 1. This apparatus outputs a new texture imagebased on the new multi-dimensional texture output from the newmulti-dimensional texture generator 104, and can realize effectivetexture mapping to a three-dimensional object.

FIG. 3 shows a process executed by the multi-dimensional texturesynthesis apparatus according to the first embodiment.

At first, a reference multi-dimensional texture is generated based onthe reference texture image according to a processing shown in FIG. 4(step 301). The reference multi-dimensional texture is quantized by thereference multi-dimensional texture quantizer 102 (step 302). A newindex image is synthesized based on the quantized referencemulti-dimensional texture (step 303). A new multi-dimensional texture ismade of the new index image and the codebook (step 304). The newmulti-dimensional texture is converted into a new texture image (step305).

FIG. 4 shows a process of the reference multi-dimensional texturegeneration step (301) executed by the reference multi-dimensionaltexture generator 101. One or more reference texture images captured ordrawn with a plurality of different conditions are input to thereference multi-dimensional texture generator (sub-step 401). In thepresent embodiment, actual captured images acquired at a plurality ofcamera positions (referred to as viewpoints) and light source positionsare used.

As mentioned above, the present embodiment directs to expressingrealistically the optical characteristic of the object surface thatvaries according to a viewpoint direction or a light source direction.The reference texture images are acquired as shown in FIG. 5.

In other words, assuming a spherical coordinate system that a normalvector of the object surface is an axis and the object surface becomesan equatorial plane (a mask passing through the equator). In thiscoordinate system, when a polar angle θ and an azimuthal angle φ areused, multi-viewpoint/multi-light source texture images to be preparedas input data can be specified by four parameters of a viewpointdirection (θE, φE) and a light source direction (θI, φI).

In the present embodiment, a texture image acquired in a certainspecific viewpoint and light source is identified by parameters of T(θE, φE, θI, φI). In the present embodiment, assuming that samples inthe viewpoint direction are W samples with respect to an axis θE and Xsamples with respect to an axis φE, and samples in the light sourcedirection are Y samples with respect to an axis θI and Z samples withrespect to an axis φI. Therefore, (W×X)×(Y×Z) texture images acquired bycombination of W×X viewpoints and Y×Z light sources are input to thereference multi-dimensional texture generator 101 as reference textureimages.

The reference multi-dimensional texture is generated by the referencemulti-dimensional texture generator 101 according to the followingprocedure.

A pixel (i, j) of the reference texture image is referred (402), and thecolor information item of the referred pixel (i, j) is extracted fromeach of all reference texture images (403).

FIG. 7 is a diagram to explain extraction of color information items ofthe same-position pixels from a plurality of reference texture images.The color components of the same-position pixels from the (W×X)×(Y×Z)reference texture images are extracted. The extracted color componentsare arranged in a given format to generate multi-dimensional pixel datafor each of the R, G and B components (404).

In other words, the color components of the same-position pixelsextracted from the plurality of reference texture images, respectively,are inserted in (W×X)×(Y×Z) pixels sequentially to generatemulti-dimensional pixel data.

The generated multi-dimensional pixel data is stored in the codebook,and an index is assigned to the referred pixel (i, j) (405). When theabove-mentioned process is done for all pixels (406), the referencemulti-dimensional texture generation process is finished. In otherwords, the sub-steps 402-406 in FIG. 4 are repeated to generate thereference multi-dimensional texture.

An example of data structure of the multi-dimensional texture is shownin FIG. 6. The multi-dimensional texture comprises a codebook and areference index image. The codebook comprises collection of data itemsas referred to as multi-dimensional pixel data (as described later), andthe reference index image comprises collection of index informationitems for addressing multi-dimensional pixel data in the codebook.

In the present embodiment, the reference index image and codebook areprepared with respect to the color component of each of R, G and Bcomponents. The multi-dimensional texture handling all of R, G and Bcomponents in a lump or the multi-dimensional texture whose brightnesscomponent and color component are separated from each other can be madein the same manner. The multi-dimensional pixel data is data obtained bycollecting color information items in certain pixels. Themulti-dimensional pixel data items of R, G and B components are assumedR (l), G (m) and B (n) respectively. These are multi-dimensional pixeldata items corresponding to indexes l, m and n, respectively. Thecodebook sizes of R, G and B components shown in FIG. 6 are L, M and N,respectively. The larger the codebook size is, the more themulti-dimensional pixel data diversifies.

FIG. 8 is a diagram for explaining a format of the multi-dimensionalpixel data according to the present embodiment. The light sourcedependence pixel data items in a specific viewpoint which are arrangedwith respect to a horizontal axis φI and a vertical axis θI are formedin a block. The block is arranged every viewpoint with respect to thehorizontal axis θE and the vertical axis φE. When all the referencetexture images are U×V pixels, the input and output of the referencemulti-dimensional texture generator 101 are as follows. Input Referencetexture Color image of 24 bits/pixel   U × V pixels   (W × V) × (Y × Z)images Output Multi-dimensional data (for each of R, G and B components)Monochrome image of 8bits/pixel   (X × Y) × (W × Z) pixels   U × Vimages Index image (for each of R, G and B components) Monochrome imageof 8bits/pixel   U × V pixels   One image

FIG. 9 is a flowchart of the reference multi-dimensional texturequantization step (302) in the reference multi-dimensional texturequantizer 102.

At first, the reference multi-dimensional texture is input to thereference multi-dimensional texture quantizer 102, and the R, G and Bcomponents are divided separately (901). When the referencemulti-dimensional texture of any one of the R, G and B components isinput (902), the quantization process is started. A parameter such as aquantization level q is prepared, and incremented from an initial value0 (903, 907). A codebook size is designated according to thequantization level q. The codebook size is increased with an increase inthe value q. The quantization process is finished at the quantizationlevel at which the codebook size equals the number of the pixels (906).An example of a method of setting the quantization level when the sizeof reference texture image is 32×32 pixels is shown as follows.Quantization level q 0 1 2 3 4 5 6 7 8 9 Codebook 2 4 8 16 32 64 128 256512 1024 size

The quantization process at each quantization level may use aconventional vector quantization (VQ). However, the present embodimentuses an improvement scheme suitable for this apparatus. The full searchvector quantization (FSVQ) adding a process to fix a cluster of error 0or the tree structured vector quantization (TSVQ) outputting a codebookof a tree structure are employed. The former scheme fixes a cluster oferror 0 as preprocessing, and does not include the fixed cluster in anobject to be subjected to a cluster division afterward. Although thisscheme is a full search, a target quantization level can be attained ata comparatively short time.

On the other hand, the latter scheme performs cluster division so as tomake a tree structure. This scheme lowers in precision of quantizationin comparison with the former scheme, but can quantize at a short time.

TSVQ has a great effect to shorten a processing time in an index imagesynthesizer described later.

After quantization, the multi-dimensional texture, namely, the referenceindex image and codebook, corresponding to each quantization level areoutput (905).

When the above-mentioned process is done for R, G and B components(908), the reference multi-dimensional texture quantization is finished.

The configuration of the quantized reference multi-dimensional textureoutput from the reference multi-dimensional texture quantizer 102 isshown in FIG. 10. The R, G and B components each hold an independentmulti-dimensional texture every quantization level. The codebook sizeincreases as q increases, and the reference multi-dimensional texturesare expressed hierarchically.

FIG. 11 is a flowchart of index image synthesis (303) in the index imagesynthesizer 103. At first, parameters are input (1101). The parametersinclude a reference texture image size, the number of the referencetexture images (size of multi-dimensional pixel data), a referencemulti-dimensional texture name (an input filename), a new texture imagesize, a new multi-dimensional texture name (an output file name), thenumber of hierarchies H, a quantization level in each hierarchy (R, G orB component), the number of scans Sh in each hierarchy, a rewrittenblock size in each scan, and a neighborhood block size in each scan.

The hierarchy number h is initialized (h=0), and an index imagesynthesis process is performed every hierarchy until the hierarchynumber becomes H−1 (1102, 1111, 1112). At each hierarchy, the referencemulti-dimensional texture of the quantization level designated by theparameter is input (1103). The designation of the quantization level ineach hierarchy is done for each of R, G and B components. Thequantization level increases as the hierarchy number increases.

Similarity of multi-dimensional pixel data existing in the inputreference multi-dimensional texture is computed (1104). This processincludes a process computing the evaluation value used when executing asimilar pixel search as described below. The index image of the inputreference multi-dimensional texture is referred, and the similarity ofmulti-dimensional pixel data designated by the index with respect tocombination of all pixels is evaluated by using SAD (Sum of AbsoluteDifference), a correlation value or an inner product as an evaluationvalue. When TSVQ is done by the reference multi-dimensional texturequantizer 102, the computation in step 1104 is not necessary, becausethe similarity can be evaluated only by tracing the codebook making atree structure.

The noise image to which the indexes existing in the reference indeximage are assigned at random is generated as an initial image of newindex image (1105). The scan number s is initialized (s=0), and theindex is rewritten by the similar pixel search as described later whilethe new index image is repeatedly scanned till the scan number becomesSh−1 (1106, 1109, 1110). In synthesis of the new index image in eachscan (1107), the new index image is rewritten every designated blocksize from the upper left to the lower right.

The similar pixel search is used for rewriting the index. The similarpixel search is to use blocks adjacent to a block to be rewritten andfind a block with the highest similarity from blocks of the same shapeexisting in the reference index image.

All blocks of the same shape existing in the reference index image maybe evaluated, or candidates may be listed from the beginning. There is amethod wherein the blocks corresponding to the positions that respectivepixels of the neighborhood block refer to are used as candidates, andthe most suitable block is selected from the candidates. When the numberof pixels of the neighborhood block is 20, the candidates are 20, sothat the searching can be efficiently done.

The similar pixel search uses not only data of the current hierarchy hbut also data of the hierarchy h−1 just before the current hierarchy. Byusing the hierarchy just before the current hierarchy, a process at thejust before hierarchy can be reflected, so that accuracy improves.

FIGS. 12A and 12B are diagrams of explaining how to select blocks in thesimilar pixel search.

The similar pixel search carries out either of a L-shaped search (FIG.12A) or a rectangular search (FIG. 12B). In the case of FIGS. 12A and12B, the to-be-rewritten block 120 has a size of 3×3 pixels and theneighborhood block 121 has a size of 7×7 pixels. The rectangular searchcarries out the search using all of 7×7 pixels of the neighborhoodblock. In contrast, the L-shaped search performs the search using pixels(20 pixels arranged in L-shape) which were already rewritten among 7×7pixels of the neighborhood block. In the rectangular search of thepresent embodiment, a block 122 formed of a rewritten block 1221 and anon-rewritten block 1222 is used. A block 123 is a to-be-rewrittenblock.

FIG. 13 is a diagram for explaining the similar pixel search for thehierarchy h. When h=0, the L-shaped search is done in the hierarchy 0 toevaluate merely by the similarity. When it is not h=0, the L-shapedsearch for the hierarchy h and the rectangular search for the hierarchyh−1 are performed. The evaluation is done by a total of thesimilarities.

In the above-mentioned process, the rewritten block size and theneighborhood block size can be changed arbitrarily every hierarchy orevery scan. As for the rewritten block size, if the rewritten block ismagnified in the sub-hierarchy, the characteristic of a rough texturecan be acquired. If the rewritten block is reduced in a higherhierarchy, the characteristic of a fine texture can be acquired. It ispreferable that the neighborhood block size to influence computationtime greatly is as small as possible. After synthesis of a new indeximage is finished by all scans carried out for all hierarchies, a newindex image is output (1108).

There will be described new multi-dimensional texture generation in thenew multi-dimensional texture generator 104 (304).

The new multi-dimensional texture generator 104 at first is suppliedwith the new index image output from the index image synthesizer 103 anda codebook with various quantization levels output from the referencemulti-dimensional texture quantizer 102. A new multi-dimensional textureis generated by combining a codebook of an arbitrary quantization levelwith a new index image synthesized in an arbitrary hierarchy. However,if the new index image coincides with the codebook in the quantizationlevel, they are combined with each other. If they are not coincide, itis necessary to rewrite the new index image in conformity with thequantization level of the codebook. In the case of rewriting, whichpixel of the reference index image is referred to is checked, so thatthe new index image is corresponded with an index in the otherquantization level.

By making the new multi-dimensional texture generator 104 output newmulti-dimensional textures of various quantization levels, it ispossible to generate various texture images from an clear new textureimage of clear detail to a coarse new texture image which seems to havereproduced only a rough pattern.

There will be described a new texture image generation of the newtexture image converter 105 (305).

At first, a new multi-dimensional texture and a viewpoint/light sourcecondition of the new texture image to be generated are input to the newtexture image converter 105. A new texture image is drawn by referringto the codebook based on an index in the index image every pixel andextracting color information of a designated viewpoint/light sourcecondition. After all pixels have been drawn, a new texture image isoutput.

The output new texture image can be applied to texture mapping tothree-dimensional object. The normal of each polygon of thethree-dimensional object and a relation between a viewpoint directionand a light source may be computed and the new texture image nearest tothe corresponding viewpoint/light source may be mapped. Interpolationusing a plurality of new texture images can generate a more realisticimage. However, it is not necessary to always generate the new textureimage to draw a pixel in units of a polygon. It is possible to draw thepixel while extracting color information of a pixel from the newmulti-dimensional texture. Texture mapping using the newmulti-dimensional texture permits an effective use of a memory.

As described above, the target index image is generated by comparing thereference multi-dimensional texture with the target multi-dimensionaltexture in the codebook.

A purpose to use a target multi-dimensional texture is to give a hint tobring as close to the pattern and raw material feeling that a user hopesas possible when synthesizing a new multi-dimensional texture largerthan a reference multi-dimensional texture from the referencemulti-dimensional texture. Accordingly, the target multi-dimensionaltexture does not have to be always acquired under many conditions,unlike the reference multi-dimensional texture.

In the capture condition shown in FIG. 5, for example, even if thereference multi-dimensional texture is captured at intervals of 20degrees in θ, the target multi-dimensional texture may be captured atintervals of 40 degrees in θ. In order words, it is how to use a targetmulti-dimensional texture to create a large multi-dimensional textureafter synthesis with a little number of samples. Accordingly, the arraysof multi-dimensional pixel data included in the codebooks of thereference multi-dimensional texture and target multi-dimensional textureoften differ from each other. Therefore, in the case of comparisonbetween the multi-dimensional pixel data, it is necessary to extractparts of identical acquisition condition, and evaluate similaritytherebetween.

Further, the reference multi-dimensional texture and targetmulti-dimensional texture do not need to be always the same material. Inthat case, a filtering process of codebook is necessary in a pre-stagefor evaluating similarity between the codebooks. For example, assumethat a red sweater of a check pattern is prepared as a referencemulti-dimensional texture, and a blue sweater of polka-dot pattern isprepared as a target multi-dimensional texture. The polka dot pattern isintended as a target, but the color of material is completely different.If the red part of the reference multi-dimensional texture should beassigned to the blue part of the target multi-dimensional texture, apre-process to convert the blue part of the target multi-dimensionaltexture into red is necessary.

Second Embodiment

FIG. 14 shows a configuration of a multi-dimensional texture synthesisapparatus according to the second embodiment of the present invention.

This apparatus adds a process using a target texture to the apparatus ofthe first embodiment. A reference multi-dimensional texture generator1401 is similar to the reference multi-dimensional texture generator 101used in the first embodiment. A reference multi-dimensional texturequantizer 1403 adds, to the reference multi-dimensional texturequantizer 102 used in the first embodiment, a process of outputting acodebook to a target index image generator 1404.

An index image synthesizer 1405 has a configuration similar to the indeximage synthesizer 103 of the first embodiment, but is modified so as touse a target index image in the similar pixel search. A newmulti-dimensional texture generator 1406 and a new texture imageconverter 1407 have configurations similar to the new multi-dimensionaltexture generator 104 and the new texture image conversion 105 that areshown in the first embodiment, respectively.

The present embodiment differs from the first embodiment in existence ofthe target multi-dimensional texture generator 1402 and the target indeximage generator 1404. The target multi-dimensional texture generator1402 generates a target multi-dimensional texture from one or moretarget texture images similarly to the reference multi-dimensionaltexture generator 1401. The target index image generator 1404 generatesa target index image so as to correspond to the codebook output from thereference multi-dimensional texture quantizer 1403.

FIG. 15 shows a configuration of a multi-dimensional texture synthesisapparatus concerning a modification of the second embodiment. In thesecond embodiment, like reference numerals are used to designate thestructural elements corresponding to those in the embodiment of FIG. 14and any further explanation is omitted for brevity's sake.

This apparatus has a configuration removing the new texture imageconverter 1407 from the apparatus of FIG. 14, and is used in the samemanner as the multi-dimensional texture synthesis apparatus shown inFIG. 2 in the first embodiment. In other words, a new texture image isoutput as a final output image.

FIG. 16 shows a series of steps (1601, 1602, 1603, 1604, 1605, 1606 and1607) of a process executed by the multi-dimensional texture synthesisapparatus according to the second embodiment shown in FIG. 14. Only thesteps that differ from those of the first embodiment are describedhereinafter.

In the present embodiment, one or more target texture images that arecaptured or drawn with different conditions are input to the targetmulti-dimensional texture generator 1402. The size of the target textureimage assumes to be equal to a required size of the new texture image.

The target multi-dimensional texture generator 1402 generates a targetmulti-dimensional texture according to a flowchart similar to that ofFIG. 4. However, the format of multi-dimensional pixel data assumes tobe similar to that of the reference multi-dimensional texture generator1401. Color information of each of target texture images to be targetedat the positions of the viewpoint and light source is written in each ofmulti-dimensional pixel data corresponding to the positions of theviewpoint and light source, respectively. The data is not written at thepositions of the viewpoint and light source not to be targeted.

To the target index image generator 1404 is input at first the targetmulti-dimensional texture and the codebook generated by the referencemulti-dimensional texture quantizer 1403. Next, the codebook of thetarget multi-dimensional texture is substituted with the codebook of thereference multi-dimensional texture, and the index is rewritten. Whenthe index is rewritten, the multi-dimensional pixel data generated bythe target multi-dimensional texture generator 1402 is compared with themulti-dimensional pixel data in the codebook to assign an indexcorresponding to the multi-dimensional pixel data of the highestsimilarity. However, the similarity is evaluated by a SAD (Sum ofAbsolute Difference), a correlation value or an inner product withoutincluding the positions of the view point and light source not to betargeted.

The above process is done for a plurality of codebooks according toquantization levels input from the reference multi-dimensional texturequantizer 1403 to output a target index image according to eachquantization level. The index image synthesizer 1405 performs a processto input a target index image and a similar pixel search process usingthe target index image as well as the process similar to that of theindex image synthesizer 103 shown in the first embodiment.

FIG. 17 is a diagram for explaining a similar pixel search for thehierarchy h. The target index image uses an image of a quantizationlevel in the hierarchy h. When h=0, the rectangular search is performedwith the hierarchy 0 to evaluate by merely the similarity.

In other words, evaluation of the similarity is done by a total ofsimilarity {circle around (1)} between a block of the new index imageand a block of the reference index image and similarity {circle around(3)} between a block of the target index image and a block of thereference index image.

When it is not h=0, the rectangular search is performed in the hierarchyh and the hierarchy h−1 so that the similarity is evaluated by a totalof the similarities, that is, similarities {circle around (1)}, {circlearound (2)} and {circle around (3)}. The hierarchy h is subjected to therectangular search (b) as shown in FIG. 12B.

The rewritten block 1221 in the rectangular block uses a block of thenew index image. The non-rewritten block 1222 uses the target indeximage. The hierarchy h-1 is subjected to the rectangular search (a) asshown in FIG. 12B. The other process is similar to that of the firstembodiment.

As described above, the target index image is generated by comparing thereference multi-dimensional texture with the target multi-dimensionaltexture in the codebook.

A purpose to use a target multi-dimensional texture is to give a hint tobring as close to the pattern and raw material feeling that a user hopesas possible when synthesizing a new multi-dimensional texture largerthan a reference multi-dimensional texture from the referencemulti-dimensional texture. Accordingly, the target multi-dimensionaltexture does not have to be always acquired under many conditions,unlike the reference multi-dimensional texture.

In the capture condition shown in FIG. 5, for example, even if thereference multi-dimensional texture is captured at intervals of 20degrees in θ, the target multi-dimensional texture may be captured atintervals of 40 degrees in θ. In order words, it is how to use a targetmulti-dimensional texture to create a large multi-dimensional textureafter synthesis with a little number of samples. Accordingly, the arraysof multi-dimensional pixel data included in the codebooks of thereference multi-dimensional texture and target multi-dimensional textureoften differ from each other. Therefore, in the case of comparisonbetween the multi-dimensional pixel data, it is necessary to extractparts of identical acquisition condition, and evaluate similaritytherebetween.

Further, the reference multi-dimensional texture and targetmulti-dimensional texture do not need to be always the same material. Inthat case, a filtering process of codebook is necessary in a pre-stagefor evaluating similarity between the codebooks. For example, assumethat a red sweater of a check pattern is prepared as a referencemulti-dimensional texture, and a blue sweater of polka-dot pattern isprepared as a target multi-dimensional texture. The polka dot pattern isintended as a target, but the color of material is completely different.If the red part of the reference multi-dimensional texture should beassigned to the blue part of the target multi-dimensional texture, apre-process to convert the blue part of the target multi-dimensionaltexture into red is necessary.

Third Embodiment

The present embodiment is a modification of the index image synthesizerused for the first and second embodiments. The multi-dimensional texturesynthesis apparatus of the present embodiment has a configurationapproximately similar to the apparatus of FIG. 1 as described in thefirst embodiment or the apparatus of FIG. 14 as described in the secondembodiment. The other components except for the index image synthesizerhave the same functions as those of the first embodiment or the secondembodiment.

FIG. 18 is a flowchart of the index image synthesis carried out by theindex image synthesizer. At first, parameters are input to the indeximage synthesizer concerning the present embodiment (1801). Because thisapparatus does not scan every hierarchy, only one scan may be set as aparameter. The parameters other than the above parameter are similar tothose of the first embodiment or the second embodiment.

The reference index images of all the hierarchs and a quantization errorare input (1802). In this apparatus, two or more reference index imageswhose quantization errors differ to each other are used to synthesize asingle new index image. The quantization error of each hierarchy is usedfor performing approximate similarity evaluation. When there is a targetindex image, the target index image also is input.

Next, the initial image of the new index image is generated (1803). Thisis a noise image similar to that of the first embodiment or the secondembodiment.

The scan number s is initialized (s=0). Index rewriting is performed bythe similar pixel search (described hereinafter) while the new indeximage is scanned repeatedly till the scan number becomes S−1 (1804,1807, 1808).

In synthesis of the new index image in each scan (1805), the new indeximage is rewritten from the upper left to the lower-right everydesignated block size. In rewriting of the index, the similar pixelsearch is performed referring to the most significant one (the smallestquantization error) of the input reference index images. The searchingmethod is similar to that of the first embodiment and the secondembodiment, but an evaluation method differs from them. In the similarpixel search in the present embodiment, the evaluation is performedusing hierarchies of two or more reference index images of differentquantization errors.

An example of similarity evaluation of pixels is shown in FIG. 19.

In order to evaluate the similarity between a block (L-shaped orsquare-shaped block) adjacent to a to-be-rewritten block and a block ofthe same shape of a reference index image or a target index image, theindex image is referred to in order from the reference index image ofthe higher hierarchy or the target index image.

When the indexes of certain pixels of two blocks which are compared witheach other coincide for the first time at a certain hierarchy, thequantization error of the hierarchy assumes to be an approximate errorin that pixel. Such an approximate error is acquired for all pixels ofthe block to evaluate the similarity between the blocks. When synthesisof the new index image in each scan is completed, the new index image isoutput (1806).

Fourth Embodiment

According to a multi-dimensional texture synthesis apparatus accordingto the first embodiment shown in FIG. 20, a group of texture imagescaptured or drawn in a plurality of different conditions is input.

A quantization process, a synthesis process, a transform process, and soon are subjected to the group of texture images to output a textureimage. According to this apparatus, a multi-dimensional texturegenerator 1601 derives image information of pixels in the same positionand gathers them to generate a multi-dimensional texture. Themulti-dimensional texture generator 101 1601 is connected to amulti-dimensional texture data storage unit 102 to store the generatedmulti-dimensional texture therein.

This multi-dimensional texture data storage unit 1602 is connected to amulti-dimensional texture quantizer 1603, an index image synthesis unit1604 and a texture image generator 1606. The multi-dimensional texturequantizer 1603 subjects the multi-dimensional texture derived from themulti-dimensional texture data storage unit 1602 to vector quantizationaccording to one or more quantization levels, and stores the quantizedmulti-dimensional texture in the multi-dimensional texture data storageunit 1602. An index image synthesis unit 1604 repeats similar pixelsearch according to control data or control image data using themulti-dimensional texture of a plurality of materials derived from themulti-dimensional texture data storage unit 1602, to synthesize a newindex image of an arbitrary size and an option design. The texture imagegenerator 1606 converts the multi-dimensional texture derived from themulti-dimensional texture data storage unit 1602 to one or more textureimages corresponding to a specific acquisition condition or creationcondition.

A generation method of multi-dimensional texture, data structure ofmulti-dimensional texture, a quantization method of multi-dimensionaltexture and an index image synthesis method are fundamentally similar tothe previous embodiment.

FIG. 21 shows a process carried out with the multi-dimensional texturesynthesis apparatus shown in FIG. 20. The process will be explainedhereinafter.

A texture group input to the multi-dimensional texture generator 1601 ofthis apparatus is acquired with various conditions (a viewing point anda light source) as explained referring to FIG. 23. In the presentembodiment, (W×X)×(Y×Z) texture images acquired by combination of all ofW×X viewing points and Y×Z light sources are input to the referencemulti-dimensional texture generator 1601 as reference texture images.

The reference multi-dimensional texture generator 1601 generates amulti-dimensional texture based on the group of the input texture images(step 201). In this case, the multi-dimensional texture generator 1601generates a multi-dimensional texture according to a procedure shown inFIG. 22 similar to FIG. 4. When a texture image set is input in step401, a pixel (i, j) of each of the reference texture images is referredto (step 402). The color information of the referred pixel (i, j) isextracted from all of the reference texture images (step 403).

FIG. 23 shows a state that extracts color information of pixels locatedin the same position from a plurality of reference texture images of thetexture image set and generates multi-dimensional pixel data. Accordingto this, from (W×X)×(Y×Z) reference texture images are derived colorcomponents of pixels located in the same position. The color componentsare gathered to a single data in conformity with a given format, and themulti-dimensional texture data is generated for each of R, G and B (step404). That is to say, the color components of pixels located in the sameposition which are extracted from a plurality of reference textureimages respectively are pasted to (W×X)×(Y×Z) pixels sequentially togenerate multi-dimensional pixel data. This process is similar to thereference multi-dimensional texture generation method explained in theprevious embodiment. However, in this embodiment, whenever themulti-dimensional texture of one material is generated, it can be storedin the multi-dimensional texture data storage unit 1602 resulting inpermitting to generate multi-dimensional textures of a plurality ofmaterials.

The generated multi-dimensional pixel data are stored in a codebook(step 405), and an index is assigned to the remarkable pixel (i, j)(step 406). Then, all pixels are remarked (step 407). When the processcompletes for all pixels, the multi-dimensional texture generationprocess is ended.

FIG. 24 shows a basic data structure of multi-dimensional texture. Themulti-dimensional texture comprises a codebook and a reference indeximage. The codebook is configured by collection of data as referred toas multi-dimensional pixel data (as described later). The referenceindex image is configured by collection of index information todesignate multi-dimensional pixel data in the codebook.

In the present embodiment, the reference index image and codebook arecreated for the color components of R, G and B. A multi-dimensionaltexture handling all of R, G and B components for a lump and amulti-dimensional texture that a luminance component and a colorcomponent are separated can be created in the same manner. Themulti-dimensional pixel data is one obtained by collecting colorinformation in a pixel. Assuming that the multi-dimensional pixel dataof R, G and B components are R(l), G(m) and B(n), respectively. Theseare multi-dimensional pixel data corresponding to indexes 1, m and n,respectively. Only one codebook is shown in FIG. 24. However, codebooksof R, G and B components are prepared as described in the previousembodiment. The codebook sizes of R, G and B components are L, M and N,respectively. The multi-dimensional pixel data increasingly diversifiesas the codebook size increases.

FIG. 25 shows a format example of multi-dimensional pixel data. In otherwords, FIG. 25 shows a format made by blocking light source dependantpixel data in a specific viewing point that are arranged in quadratureaxis θI and ordinate axis φI, and arranging the blocks in quadratureaxis φE and ordinate axis θE every view point. The data structure ofmulti-dimensional texture and the format of multi-dimensional pixel dataare similar to the previous embodiment.

FIG. 26 shows a procedure of multi-dimensional texture quantization inthe multi-dimensional texture quantizer 103 (202). According to this, atfirst a reference multi-dimensional texture is input (801). Aquantization level q is initialized (802) and quantization is started inthe quantization level q (803). An index image and a codebook in thequantization level q are output (804). In quantization level, thecodebook size is specified. In this case, the codebook size increaseswith increase of the value q. In step 805, a quantization error iscomputed and output.

The quantization process completes in a quantization level that thecodebook size equals to the number of pixels of the reference textureimage (806). When q<Q is “No”, the quantization level is changed orincremented by 1 (807).

FIG. 27 shows configuration of the quantized reference multi-dimensionaltexture output from the multi-dimensional texture quantizer 103. The R,G and B components hold respective reference multi-dimensional texturesevery quantization level, distinctly. The codebook size increases as qincreases. The reference multi-dimensional texture is expressed inhierarchy.

The above process is similar to that of the reference multi-dimensionaltexture quantizer described by the previous embodiment. However, in thepresent embodiment, the multi-dimensional texture is derived from themulti-dimensional texture data storage unit 102. The process for storingthe quantized multi-dimensional texture in the multi-dimensional texturedata storage unit 102 can repeated. For this reason, themulti-dimensional textures of a plurality of materials can be quantized.

FIG. 28 shows a procedure of index image synthesis (203) in the indeximage synthesizer 104. According to this, at first a plurality ofmulti-dimensional textures are input (1001). A parameter, for example, asearch block size, the number of scanning times s are input (1002). Aninitial image of a new index image is generated (1003). In other words,a noise image to which indexes existing in the reference index image arerandomly assigned is generated as the initial image of new index image.

The scanning number s is initialized (s=0) (1004), and a control imageis input (1005). A new index image is synthesized by a similar pixelsearch using correlation of multi-dimensional pixel data or aquantization error (1006). The new index image in the scanning number sis output (1007). It is determined whether or not s<S (step 1008). Ifs<S, the scanning number s is changed as s=s+1 (1009). Then, steps1005-1007 are repeated until s<S is not concluded.

In other words, L-shaped block as shown in FIG. 29 is used for a similarpixel search to rewrite an index as described below while the new indeximage is scanned repeatedly till the scanning number s becomes not morethan S. For an evaluation value in the similar pixel search is used anapproximate evaluation value dependent on a quantization error, by usingSAD (Sum of Absolute Difference) between the multi-dimensional pixeldata or a hierarchy of a multi-dimensional texture corresponding to aquantization level as shown in FIG. 30 is used.

Information included in the control image input to the index imagecomposition unit 104 is shown in FIG. 31. The control image is a coloredimage expressing control data of each pixel at 24 bits. FIG. 31 is anexample referring to three kinds of material. A probability that eachmaterial is selected every 8 bits or a weighting factor is included inthe control image. The imaging of the control data provides a merit thata user (a designer) can visualize a synthesis result and a merit that animage can be controlled with such sense to sketch an object. The controldata of each pixel includes a material ID to specify a material to bereferred in a similar pixel search, a weighted parameter to specify howmuch probability each material is selected, when a plurality ofmaterials are referred to, or a selection probability.

FIG. 32 shows how each material is referred to according to the controlimage in drawing an index by a similar pixel. The material is specifiedso that a material 1 is referred to in the upper part of the image, amaterial 2 in the central part thereof, and a material 3 in the lowerpart thereof. In a boundary part between materials 1 and 2, bothmaterials are referred to so that the boundary between materials issmoothly connected without sense of incongruity. The material isspecified selectively so as to carry out a process of drawing an index.

In drawing a pixel, when an index of a material different from amaterial to be referred exists in an index of a block neighborhood tothe pixel, some artifices are needed. One of the artifices is shown inFIG. 33. In FIG. 33, indexes of materials 1 and 2 are drawn in a blockneighborhood to a pixel to be drawn.

According to the control image, two materials 1 and 2 must be referredto. When referring to the material 1, there is carried out a process totemporarily replace the part that an index of the material 2 of theneighborhood block is drawn with an index of the material 1. When thereplacement is executed, it may be replaced with an index of thematerial 1 with highest similarity or may be replaced randomly.

Alternatively, the multi-dimensional pixel data of different materialsin the codebook are compared with each other by using the index as it iswithout doing such a replacement process. The evaluation can be carriedout by means of SAD. When referring to the material 2, the similarmethod can be applied. After having extracted, referring to thematerials 1 and 2, an optimum pixel corresponding to each of thematerials, weighting of an error or selection due to selectionprobability and so on is carried out. When referring to materials morethan three kinds, the similar method is applied.

In this case, it is not necessary to scan the new index image simplyfrom end to end thereof. For example, it is possible to searches for atfirst a pixel which refers to only one material surely by a similarpixel search, and then draw a pixel which must refer to a plurality ofmaterials. In this approach, control becomes difficult. However, it ispossible to use a pixel which has been drawn surely by reference of onematerial as one part of a neighborhood block. Therefore, it is possibleto generate a texture which is near to a user's request by a few scans.

In addition, if a user can give a control factor whenever a new indeximage is drawn by one scanning, an interactive synthesis can berealized.

Generation (204) of texture image by the texture image generator 1051606 is the same as that of the new texture image converter described bythe previous embodiment. The present embodiment extracts amulti-dimensional texture from the multi-dimensional texture datastorage unit 1602 and derives a multi-dimensional texture of a pluralityof materials to generate a texture image with a variation is different.This differs from the previous embodiment.

The texture image generator 1606 extracts a multi-dimensional texturefrom the multi-dimensional texture data storage unit 1602 and provides aview point and light source condition of the texture image to begenerated. The codebook is referred to on the basis of an index in anindex image every pixel, and color information of a specified view pointand light source condition is derived to draw a texture image. After allpixels were drawn, a texture image set is output.

The output texture image set can be applied to texture mapping for athree-dimensional object. A relation between a normal line of eachpolygon of the three-dimensional object and a view point direction and alight source direction may be computed, and a texture image which isnearest to a corresponding view point and light source may be mapped. Amore realistic image can be generated by interpolating using a pluralityof texture images. However, it is not necessary to always generate thetexture image to draw a pixel in a polygon unit. The drawing can becarried out while extracting color information of the pixel from themulti-dimensional texture. A memory can be effectively utilized bytexture-mapping using the multi-dimensional texture.

Further, a user can use specific multi-dimensional texture data, andprocess and store the same by publicly opening on a network data of themulti-dimensional texture data storage unit 1602 as database. A group oftextures that a certain user acquired or created is converted into aformat of multi-dimensional texture data and publicly opened. Then,another user can synthesizes index image using the multi-dimensionaltexture data, and apply to texture mapping. Further, a further anotheruser can reuse the multi-dimensional texture data by publicly opening ona network the multi-dimensional texture data subjected to the indeximage synthesis.

Fifth Embodiment

A multi-dimensional texture synthesis apparatus according to the presentembodiment shown in FIG. 34 adds a multi-dimensional texture converter1605 to the multi-dimensional texture synthesis apparatus (FIG. 20) ofthe first embodiment. In the embodiment, like reference numerals areused to designate like structural elements corresponding to those in theembodiment of FIG. 20 and any further explanation is omitted forbrevity's sake. An operation of the multi-dimensional texture synthesisapparatus of the present embodiment will be described referring to FIG.35. In step 1701, a multi-dimensional texture generator 1601 generates amulti-dimensional texture based on a group of input texture images. Instep 1702, the multi-dimensional texture quantizer 1603 quantizes eachmulti-dimensional texture according to the same procedure as the firstembodiment. In step 1703, as a preprocessing of a synthesis (step 1704)of an index image according to the control image, a multi-dimensionaltexture converter 1605 extracts a multi-dimensional texture from amulti-dimensional texture data storage unit 1602. Then, themulti-dimensional texture converter 1605 subjects the multi-dimensionalpixel data to a blending process and a filtering process. The processedmulti-dimensional texture is stored in a multi-dimensional texture datastorage unit 1602.

An example of the blending process is shown in FIG. 36. For example,when two kinds of multi-dimensional textures are blended, themulti-dimensional textures of materials 1 and 2 are input to themulti-dimensional texture converter 1605 to compare two index images.New multi-dimensional pixel data are generated by calculating an averageof multi-dimensional pixel data of material 1 designated by an index inthe same pixel position and multi-dimensional pixel data of material 2or adding two multi-dimensional pixel data based on a blend rate. Thenew multi-dimensional pixel data is registered in a new codebook, and anindex image is generated newly. Such a process can generate anintermediate image in carrying out morphing between materials 1 and 2.The similar process can do for materials more than three kinds.

Various filtering processes such as changing of luminance or changing oftransmittance are subjected to the multi-dimensional texture of onematerial, and to specific pixel or all pixels or specific or all viewpoints/source positions. It is possible by this filtering process toobtain texture image subjected to multi-dimensional image processingwithout captured or drawn texture image itself, and further to give aspecial effect to the texture image. It is possible to provide an effectthat a kind of an irradiated light source changes according to aposition of the texture or an effect such as fades of a material andexpansion and contraction of the material.

When a plurality of codebooks are referred to by a certain index imageas shown in FIG. 37, it is possible to carry out a process to bringtogether the codebooks. In FIG. 37, since an input index image refers tothe codebook of materials 1, 2 and 3, a plurality of materials ID andaddress information of the codebook are included in an index. Therefore,administration and operation are difficult. Consequently, themulti-dimensional pixel data in referring is brought together to onecodebook to generate an index image newly. This makes it possible torecycle the index image as a new material in the index synthesisapparatus and generate easily a texture image in the texture imagegenerator 1606.

The multi-dimensional texture conversion processing described above canuse for a preprocessing (step 1703) or a postprocessing (1705) of asynthesis process of index image in the index image composition unit1604, resulting in permitting to process the multi-dimensional texturewith freedom. Generation (1706) of texture image by the texture imagegenerator 1606 is the same as that of the new texture image converterdescribed by the previous embodiment.

According to an embodiment of the present invention as mentioned above,the use of a texture image varying with conditions such as view pointdirection or light source direction permits synthesizing of a newtexture image of an arbitrary size while taking control from a user.According to the apparatus of the first embodiment, a multi-dimensionaltexture having an arbitrary pattern or a design can be synthesized usingthe multi-dimensional textures of a plurality of materials. According tothe apparatus of the second embodiment, an intermediate image ofmorphing or a multi-dimensional texture with a special effect can besynthesized by a blending process of multi-dimensional pixel data or afiltering processes.

In addition, the multi-dimensional texture acquired or made once or themulti-dimensional texture quantized, synthesized and converted with thisapparatus can be reused by storing a plurality of multi-dimensionaltexture in the data storage unit. In other words, it is possible togenerate various multi-dimensional textures in good efficiency.

According to the embodiment of the present invention as mentioned above,synthesizing a new texture image of an arbitrary size can be carried outusing the texture image varying by a condition such as a viewpointdirection or a light source direction.

According to the apparatus described in the first embodiment, the imageobtained by reproducing a characteristic of the reference texture imageas it is can be synthesized.

According to the apparatus described in the second embodiment, a patternor a design similar to the target texture image is maintained, and it ispossible to synthesize such a new texture image as to reproduce thecharacteristic of the reference texture image.

According to the apparatus described in the third embodiment, thecomputation using the multi-dimensional pixel data can be omitted asmuch as possible. As a result, the third embodiment is effective in adata amount and a computation time, and can maintain synthesisprecision.

According to data structure of the multi-dimensional texture, themulti-dimensional texture image data can be effectively held, so thateffective texture mapping can be realized.

It is possible to use multimedia data such as one or more images,pictures, speeches that are captured or drawn with a plurality ofdifferent conditions and apply the present invention to an apparatus tosynthesize multimedia data of an arbitrary size or an arbitrary timelength corresponding to each of the conditions.

According to the present invention, there can be provided amulti-dimensional texture synthesis apparatus and method that caneffectively synthesize from a large quantity of texture image data atexture image of an arbitrary size which can express realistically theoptical characteristic of an object surface varying according to aviewpoint direction and a light source direction.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. A multi-dimensional texture synthesis apparatus to synthesize a newmulti-dimensional texture using one or more texture images, comprising:a reference multi-dimensional texture generator receiving, as one ormore reference texture images, one or more texture images captured orcreated with a plurality of different conditions to generate a referencemulti-dimensional texture including a codebook and a reference indeximage, the codebook including a set of color information ofsame-position pixels of one or more reference texture images, and thereference index image including a set of indexes for addressing thecodebook; a reference multi-dimensional texture quantizer to quantizethe reference multi-dimensional texture at one or more quantizationlevels and generate a quantized codebook and a quantized reference indeximage; an index image synthesizer to synthesize a new index image havinga specified size from the quantized reference index image using ahierarchical structure of the reference multi-dimensional texturecorresponding to one of the quantization levels; and a newmulti-dimensional texture generator to generate a new multi-dimensionaltexture by combining the new index image with the quantized codebook. 2.The multi-dimensional texture synthesis apparatus according to claim 1,wherein the reference multi-dimensional texture generator generates thereference index image by extracting the color information items of thesame-position pixels from all the reference texture images, arrangingthe color information in a given format to generate multi-dimensionalpixel data, storing the multi-dimensional pixel data in the codebook,and assigning indexes to the pixels.
 3. The multi-dimensional texturesynthesis apparatus according to claim 1, wherein the referencemulti-dimensional texture quantizer quantizes the reference highdimensional multi-dimensional texture corresponding to the one or morequantization level by subjecting the reference multi-dimensional textureto vector quantization at the quantization level determined dependentupon a size of the codebook.
 4. The multi-dimensional texture synthesisapparatus according to claim 3, wherein the vector quantization is afull search vector quantization including a process for fixing a clusterof error to
 0. 5. The multi-dimensional texture synthesis apparatusaccording to claim 3, wherein the vector quantization is atree-structured vector quantization including a process for searchingfor similarity of the multi-dimensional pixel data according to a treestructure configuring the codebook.
 6. The multi-dimensional texturesynthesis apparatus according to claim 1, wherein the index imageincludes a Plurality of blocks each having a specified size, and theindex image synthesizer inputs the reference multi-dimensional texturequantized at one of the quantization levels, and includes means forsynthesizing a new index image by scanning a noise image randomlyincluding the indexes contained in the reference index image to rewritethe indexes of the noise image for each of the blocks by using a similarpixel search searching the reference index image for a block having ahighest similarity with respect to a rewritten block adjacent to ato-be-rewritten block of the noise image.
 7. The multi-dimensionaltexture synthesis apparatus according to claim 6, wherein the indeximage synthesizer uses one of SAD (Sum of Absolute Difference), acorrelation value and an inner product to obtain an evaluation value ofthe similarity of the multi-dimensional pixel data in the similar pixelsearch.
 8. The multi-dimensional texture synthesis apparatus accordingto claim 6, wherein the index image synthesizer includes means forevaluating the similarity of the multi-dimensional pixel data in thesimilar pixel search by using output data of the multi-dimensionaltexture quantizer carrying out a tree structure vector quantization andtracing a tree of the codebook of a tree-structure, and the means forevaluating the similarity includes means for evaluating the similarityby means of the same tree-structure for the index image synthesis ofeach of a plurality of hierarchies.
 9. The multi-dimensional texturesynthesis apparatus according to claim 1, wherein the index imagesynthesizer evaluates, in the similar pixel search, the similarity ofthe block of the reference index image with respect to the rewrittenblock between the new index image generated at a hierarchy one beforeone from the hierarchy of the reference multi-dimensional texture andthe reference index image to obtain a last evaluation value.
 10. Themulti-dimensional texture synthesis apparatus according to claim 1,wherein the index image synthesizer carries out a similar pixel searchusing a reference index image of a lower hierarchy than the referenceindex image, and determining as an approximate evaluation value aquantization error in the hierarchy where the index of the neighborhoodblock coincides with that of the reference index image of the lowerhierarchy for a first time.
 11. The multi-dimensional texture synthesisapparatus according to claim 1, which includes a targetmulti-dimensional texture generator to generate a targetmulti-dimensional texture including a set of color information of thesame-position pixels by receiving one or more texture images as one ormore target texture images, and a target index image generator togenerate a target index image by quantizing the target multi-dimensionaltexture using the codebook, and wherein the index image synthesizer usesthe target index image.
 12. The multi-dimensional texture synthesisapparatus according to claim 11, wherein the target multi-dimensionaltexture generator includes: means for generating the targetmulti-dimensional texture image by extracting the color information ofthe same-position pixels from all of the one or more target textureimages, means for arranging the color information in a format togenerate multi-dimensional pixel data, means for storing the generatedmulti-dimensional pixel data in the codebook, means for providing anaddress of the codebook as an index information item of a pixel, andmeans for assigning a set of index information to the pixels to generatethe target multi-dimensional texture image.
 13. (canceled)
 14. Themulti-dimensional texture synthesis apparatus according to claim 11,which includes means for determining an evaluation value of similaritybetween a first block and a second block, the first block being a blockof the target index image that is at the same position as that of ablock that fails to be rewritten in blocks adjacent to a block to bewritten, and the second block being a block having the same shape as thefirst block in the reference index image.
 15. The multi-dimensionaltexture synthesis apparatus according to claim 11, wherein the indeximage synthesizer includes: means for receiving referencemulti-dimensional textures quantized at two or more quantization levelsfrom the reference multi-dimensional texture quantizer, means forgenerating new index images of plural sizes using hierarchies accordingto the quantization levels, and means for using a reference index imageof lower hierarchy than the reference index image used for synthesis ofthe new index images in a similar pixel search to determine as anapproximation evaluation value a quantization error of the hierarchywhere an index of the neighborhood block coincides with that of thelower hierarchy reference index image for the first time.
 16. Themulti-dimensional texture synthesis apparatus according to claim 1,wherein the new multi-dimensional texture generator includes means forrewriting the new index image in conformity with the quantization levelof the codebook when combining the new index image synthesized at aquantization level with the codebook of the quantization level. 17.(canceled)
 18. The multi-dimensional texture synthesis apparatusaccording to claim 1, which further comprises a new texture imageconverter to convert the new multi-dimensional texture to one or morenew texture images corresponding to a specific acquisition condition ora generation condition.
 19. The multi-dimensional texture synthesisapparatus according to claim 18, wherein the new texture image converterincludes means for reading multi-dimensional pixel data from thecodebook based on an index stored in each pixel of the new index image,and means for extracting color information corresponding to the specificacquisition condition or the generation condition from themulti-dimensional pixel data.
 20. A multi-dimensional texture synthesismethod of synthesizing a new multi-dimensional texture using one or moretexture images, comprising: inputting, as one or more reference textureimages, one or more texture images captured or drawn with a plurality ofdifferent conditions; generating a reference multi-dimensional textureincluding a codebook and a reference index image, the codebook includinga set of color information of the same-position pixels of the one ormore reference texture images, and the reference index image including aset of indexes for addressing the codebook; quantizing the referencemulti-dimensional texture at one or more quantization levels to generatea quantized codebook and a quantized reference index image; synthesizinga new index image having a specified size from the quantized referenceindex image using a hierarchical structure of the referencemulti-dimensional texture corresponding to the one or more quantizationlevels; and generating the new multi-dimensional texture by combiningthe new index image with the quantized codebook.
 21. A multi-dimensionaltexture synthesis program stored in a computer readable medium forsynthesizing a new multi-dimensional texture using one or more textureimages, comprising: means for instructing a computer to input, as one ormore reference texture images, one or more texture images acquired orcreated with a plurality of different conditions; means for instructingthe computer to generate a reference multi-dimensional texture includinga codebook and a reference index image, the codebook including a set ofcolor information of the same-position pixels of the reference textureimages, and the reference index image including a set of indexes foraddressing the codebook; means for instructing the computer to quantizethe reference multi-dimensional texture at one or more quantizationlevels to generate a quantized codebook and a quantized reference indeximage: means for instructing the computer to synthesize a new indeximage having a specified size from the quantized reference index imageusing a hierarchical structure of the reference multi-dimensionaltexture corresponding to the one or more quantization levels; and meansfor instructing the computer to generate the new multi-dimensionaltexture by combining the new index image with the quantized codebook.22. A multi-dimensional texture synthesis apparatus to generate atexture image of a specified size and a specified design using a textureimage set on one or more kinds of material, the apparatus comprising: amulti-dimensional texture generator receiving one or more texture imagescaptured or created with a Plurality of different conditions as atexture image set to generate a multi-dimensional texture including acodebook formed of multi-dimensional pixel data generated by collectingimage information of pixels in the same position from the one or moretexture images of the texture image set and an index image formed ofindex information to address the codebook; a multi-dimensional texturedata storage unit configured to store the multi-dimensional texture; anindex image synthesis unit configured to derive the multi-dimensionaltexture of one or more kinds of material from the multi-dimensionaltexture data storage unit, synthesize an index image of a specific sizebased on an index image of one or more kinds of material and accordingto control data provided by a user or a control image into which thecontrol data is converted, and store a synthesis index image in themulti-dimensional texture data storage unit; a texture image generatorconfigured to derive the multi-dimensional texture from themulti-dimensional texture data storage unit, and convert it into one ormore texture images corresponding to a specific capturing condition or aspecific generation condition, wherein the texture image of the specificdesign is generated by combination of a plurality of materials.
 23. Themulti-dimensional texture synthesis apparatus according to claim 1,which further comprises a database to store the multi-dimensionaltexture data publicly on a network.